Electronic camera and range sensor systems that provide a measure of distance z from the system to a target object are known in the art. Many such systems approximate the range to the target object based upon luminosity or brightness information obtained from the target object. Some such systems are passive and respond to ambient light reflected from the target object, while other systems emit and then detect emitted light reflected from the target object. However brightness or luminosity-based systems may erroneously yield the same measurement information for a distant target object that may be large with a shiny surface and is thus highly reflective, as for a smaller target object that is closer to the system but has a dull surface that is less reflective.
Some range finder autofocus cameras employ detectors to approximate average distance from the camera to the target object by examining relative luminosity (brightness) data obtained from the object. In some applications such as range finding binoculars, the field of view is sufficiently small such that all objects in focus will be at substantially the same distance. But in general, luminosity-based systems do not work well and are subject to error, as noted above.
Some cameras and binoculars employ active infrared (1R) autofocus systems that produce a single distance value that is an average or a minimum distance to all targets within the field of view. Other camera autofocus systems often require mechanical focusing of the lens onto the subject to determine distance. At best these prior art focus systems can focus a lens onto a single object in a field of view, but cannot simultaneously measure distance for all objects in the field of view.
In general, a reproduction or approximation of original luminosity values in a scene permits the human visual system to understand what objects were present in the scene and to estimate their relative locations stereoscopically. For non-stereoscopic images such as those rendered on an ordinary television screen, the human brain assesses apparent size, distance and shape of objects using past experience. Specialized computer programs can approximate object distance under special conditions.
Stereoscopic images allow a human observer to more accurately judge the distance of an object. However it is challenging for a computer program to judge object distance from a stereoscopic image. Errors are often present, and the required signal processing requires specialized hardware and computation. Stereoscopic images are at best an indirect way to produce a three-dimensional image suitable for direct computer use.
Many applications require directly obtaining a three-dimensional rendering of a scene. But in practice it is difficult to accurately extract distance and velocity data along a viewing axis from luminosity measurements. Other applications may require accurate distance and velocity tracking, for example an assembly line welding robot that must determine the precise distance and speed of the object to be welded, or an imaging warning system for use in a motor vehicle. The necessary distance measurements may be erroneous due to varying lighting conditions and other shortcomings noted above. Such applications would benefit from a system that could directly capture three-dimensional imagery.
Some prior art system seek to acquire three dimensional images using mechanical devices. For example, scanning laser range finding systems raster scan an image by using mirrors to deflect a laser beam in the x-axis and perhaps the y-axis plane. The angle of deflection of each mirror is used to determine the coordinate of an image pixel being sampled. Such systems require precision detection of the angle of each mirror to determine which pixel is currently being sampled. Understandably having to provide precision moving mechanical parts add bulk, complexity, and cost to such range finding system. Further, because these systems sample each pixel sequentially, the number of complete image frames that can be sampled per unit time is limited. (It is understood that the term “pixel” can refer to an output result produced from one or more detectors in an array of detectors.) Although specialized three dimensional imaging systems exist in the nuclear magnetic resonance and scanning laser tomography fields, such systems require substantial equipment expenditures. Further, these systems are obtrusive, and are dedicated to specific tasks, e.g., imaging internal body organs.
Rather than rely exclusively upon acquiring luminosity data, actively or passively, a more accurate distance measuring system is the so-called time-of-flight (TOF) system. FIG. 1 depicts an exemplary TOF system, as described in U.S. Pat. No. 6,323,942 entitled CMOS-Compatible Three-Dimensional Image Sensor IC (2001), which patent is incorporated herein by reference as further background material. TOF system 100 can be implemented on a single IC 110, without moving parts and with relatively few off-chip components. System 100 includes a two-dimensional array 130 of pixel detectors 140, each of which has dedicated circuitry 150 for processing detection charge output by the associated detector. In a typical application, array 130 might include 100×100 pixels 230, and thus include 100×100 processing circuits 150. IC 110 also includes a microprocessor or microcontroller unit 160, memory 170 (which preferably includes random access memory or RAM and read-only memory or ROM), a high speed distributable clock 180, and various computing and input/output (I/O) circuitry 190. I/O circuitry 190 includes functions such as analog-to-digital (A/D) conversion of the output signals from the pixel photodetectors in array 130, and system video gain. Among other functions, controller unit 160 may perform distance to object and object velocity calculations.
Under control of microprocessor 160, a source of optical energy 120 is periodically energized and emits optical energy via lens 125 toward an object target 20. Typically the optical energy is light, for example emitted by a laser diode or LED device 120. Some of the emitted optical energy will be reflected off the surface of target object 20, and will pass through an aperture field stop and lens, collectively 135, and will fall upon two-dimensional array 130 of pixel detectors 140 where an image is formed. Each imaging pixel detector 140 measures both intensity or amplitude of the optical energy received, and the phase-shift of the optical energy as it travels from emitter 120, through distance Z to target object 20, and then distance again back to imaging sensor array 130. For each pulse of optical energy transmitted by emitter 120, a three-dimensional image of the visible portion of target object 20 is acquired.
Emitted optical energy traversing to more distant surface regions of target object 20 before being reflected back toward system 100 will define a longer time-of-flight than radiation falling upon and being reflected from a nearer surface portion of the target object (or a closer target object). For example the time-of-flight for optical energy to traverse the roundtrip path noted at t1 is given by t1=2·Z1/C, where C is velocity of light. A TOF sensor system can acquire three-dimensional images of a target object in real time. Such systems advantageously can simultaneously acquire both luminosity data (e.g., signal amplitude) and true TOF distance measurements of a target object or scene.
As described in U.S. Pat. No. 6,323,942, in one embodiment of system 100 each pixel detector 140 has an associated high speed counter that accumulates clock pulses in a number directly proportional to TOF for a system-emitted pulse to reflect from an object point and be detected by a pixel detector focused upon that point. The TOF data provides a direct digital measure of distance from the particular pixel to a point on the object reflecting the emitted pulse of optical energy. In a second embodiment, in lieu of high speed clock circuits, each pixel detector 140 is provided with a charge accumulator and an electronic shutter. The shutters are opened when a pulse of optical energy is emitted, and closed thereafter such that each pixel detector accumulates charge as a function of return photon energy falling upon the associated pixel detector. The amount of accumulated charge provides a direct measure of round-trip TOF. In either embodiment, TOF data permits reconstruction of the three-dimensional topography of the light-reflecting surface of the object being imaged.
In some TOF systems, array 130 may comprise an array of charge-coupled devices (CCDs) or CMOS devices. CCDs typically are configured in a so-called bucket-brigade whereby light-detected charge by a first CCD is serial-coupled to an adjacent CCD, whose output in turn is coupled to a third CCD, and so on. Bucket-brigade configurations generally preclude fabricating processing circuitry on the same IC containing the CCD array. Further, CCDs provide a serial readout as opposed to a random readout. For example, if a CCD range finder system were used in a digital zoom lens application, even though most of the relevant data would be provided by a few of the CCDs in the array, it would nonetheless be necessary to readout the entire array to gain access to the relevant data, a time consuming process. In still and some motion photography applications, CCD-based systems might still find utility.
Many factors including ambient light can affect reliability of data acquired by TOF systems. In some applications the transmitted optical energy may be emitted multiple times using different systems settings to increase reliability of the acquired TOF measurements. For example, the initial phase of the emitted optical energy might be varied to cope with various ambient and reflectivity conditions. The amplitude of the emitted energy might be varied to increase system dynamic range. The exposure duration of the emitted optical energy may be varied to increase dynamic range of the system. Further, frequency of the emitted optical energy may be varied to improve the unambiguous range of the system measurements.
U.S. Pat. No. 6,580,496 entitled Systems for CMOS-Compatible Three-Dimensional Image-Sensing Using Quantum Efficiency Modulation (2003) discloses a sophisticated system in which relative phase (φ) shift between the transmitted light signals and signals reflected from the target object is examined to acquire distance z. (U.S. Pat. No. 6,515,740 issued from the same application and is a companion application to U.S. Pat. No. 6,580,496.) Detection of the reflected light signals over multiple locations in a pixel array results in measurement signals that are referred to as depth images. FIG. 2A depicts a system 100′ according to the '496 patent, in which an oscillator 115 is controllable by microprocessor 160 to emit high frequency (perhaps 200 MHz) component periodic signals, ideally representable as A·cos(ωt). Emitter 120 transmitted optical energy having low average and peak power in the tens of mW range, which emitted signals permitted use of inexpensive light sources and simpler, narrower bandwidth (e.g., a few hundred KHz) pixel detectors 140′. Unless otherwise noted, elements in FIG. 2A with like reference numerals to elements in FIG. 1 may be similar or identical elements.
In system 100′ there will be a phase shift φ due to the time-of-flight (TOF) required for energy transmitted by emitter 120 (S1=cos(ωt)) to traverse distance z to target object 20, and the return energy detected by a photo detector 140′ in array 130′, S2=A·cos(ωt+φ), where A represents brightness of the detected reflected signal and may be measured separately using the same return signal that is received by the pixel detector. FIGS. 2B and 2C depict the relationship between phase shift φ and time-of-flight, again assuming for ease of description a sinusoidal waveform. The period for the waveforms of FIGS. 2B and 2C is T=2π/ω.
The phase shift φ due to time-of-flight is:φ=2·ω·z/C=2·(2πf)·z/C
where C is the speed of light 300,000 Km/sec. Thus, distance z from energy emitter (and from detector array) to the target object is given by:z=φ·C/2ω=φ·C/{2·(2πf)}
While many types of three-dimensional TOF imaging systems are known in the art, obtaining reasonably accurate depth images from such systems can be challenging. Various techniques for acquiring and processing three dimensional imaging have been developed by assignee herein Canesta, Inc. of San Jose, Calif. For example, U.S. Pat. No. 6,522,395 (2003) to Bamji et al. describes Noise Reduction Techniques Suitable for Three-Dimensional Information Acquirable with CMOS-Compatible Image Sensor ICs; and U.S. Pat. No. 6,512,838 to Rafii et al. (2003) describes Methods for Enabling Performance and Data Acquired from Three-Dimensional Image Systems. U.S. Pat. No. 6,678,039 to Charbon (2004) discloses Method and System to Enhance Dynamic Range Conversation Useable with CMOS Three-Dimensional Imaging. More recently, U.S. Pat. No. 6,919,549 to Bamji et al. (2005) describes Method and System to Differentially Enhance Sensor Dynamic Range.
In embodiments of the above-noted TOF systems developed by Canesta, Inc., detection current output by individual pixel detector sensors was directly integrated and collected using an integration capacitor to develop a detection voltage signal. In U.S. Pat. No. 6,678,039, the integrated detection voltage signal was compared to a threshold voltage and whenever the threshold voltage was exceeded, the capacitor was reset, and the number of resets was stored. In this fashion, the total change in detection voltage could be measured, and dynamic range could be extended by virtue of the resets. In U.S. Pat. No. 6,919,549, before start of integration, a fixed potential was imposed upon each capacitor. In the '549 invention, during integration, the potential across each capacitor could be reset before the integration detection voltage signal reach saturation, or overload, levels. Again the total change in detection voltage could be measured and dynamic range could be extended by virtue of the pre-integration voltage and resets. Preferably the sensors in many of these embodiments were operable in differential mode, to better reject common mode signals such as ambient light.
As noted in many of the above-referenced patents, sensor performance may be impaired by many parameters. Some or all sensors may not receive sufficient optical energy to perform well, in which case a reliable measurement of depth z cannot be obtained. Similarly, some or all sensors may receive too much optical energy, in which case the sensors may overload and saturate, with the result that a reliable measurement of depth z cannot be obtained. It is when a sensor receives an amount of optical energy within the dynamic range of the sensor that an accurate sensor output signal is provided, and reliable measurements of depth z can be obtained.
Thus there is a need for a mechanism for use with pixel detector sensors in a CMOS three-dimensional TOF system to maximize the number of sensors that are operating within their dynamic range. Preferably such mechanism should function dynamically such that as system parameters that affect sensor performance change, appropriate compensating changes or corrections to individual pixel detector sensors can be made. The desired result is that more valid z-depth measurement data will be provided despite variations in such system parameters.
The present invention provides a method and sub-system to dynamically provide compensating changes to pixel detector sensors such that more valid z depth measurement data will be provided, despite changes in system parameters that would otherwise impair more of the z depth measurement data.